speaking out against people’s favorite toys, tech leaders who appeal to a particular type of nerdy masculinity, and exploitative practices draws all kinds of negative pushback.

Mystery AI Hype Theater

“Along the way, we learn to always read the footnotes.” That’s because checking the sources for all of the hype-tastic claims often gives us a good vista on the house of cards (that is, thin research methods, shoddy argumentation, and questionable citation practice) supporting the flashy façade.

Think of us as your guides to navigating a glitzy technology expo hall, full of salespeople trying to get you to buy a new product or fork over your data. We don’t need that energy, and neither do you.

Artificial intelligence, if we’re being frank, is a con: a bill of goods you are being sold to line someone’s pockets. A few major well-placed players are poised to accumulate significant wealth by extracting value from other people’s creative work, personal data, or labor, and replacing quality services with artificial facsimiles.

A characteristic of our current hype cycle is that the con men are taking a series of tropes from science fiction—of artificial minds hell-bent on turning us into paper clips or Terminators waging wars for their right to exist (and to look cool on motorcycles)—and injecting them into discussions at the highest echelons of business and government. This framing is useful to those creating the technology because it makes them appear powerful—if not godlike—in their technical creation. But this belies what these technologies are doing to the rest of us: threatening stable careers and replacing them with gig work, slashing personnel in government, cheapening our social services, and degrading creativity.

“AI” is a marketing term. It doesn’t refer to a coherent set of technologies. Instead, the phrase “artificial intelligence” is deployed when the people building or selling a particular set of technologies will profit from getting others to believe that their technology is similar to humans, able to do things that, in fact, intrinsically require human judgment, perception, or creativity.

every time we write “AI”, imagine we have a set of scare quotes around it. Or if you prefer, replace it with a ridiculous phrase. Some of our favorites include “mathy maths”, “a racist pile of linear algebra”, “stochastic parrots” 16 (referring to large language models specifically), or Systematic Approaches to Learning Algorithms and Machine Inferences (aka SALAMI17).

the conversation becomes clearer if one speaks in terms of “automation” rather than “AI” and looks at precisely what is being automated. In doing so, we find several types of automation.

Decision making. The first group involves using computers to automate consequential decisions. These are called automatic decision systems and they are often used, for example, in the process of setting bail, approving loans, screening résumés, or allocating social benefits. These uses are contentious, and rightfully so, because they have extreme ramifications for people who are subject to the system’s recommendations.

Classification. The second kind of automation involves classification of inputs of different types. For example, image classification can be used to help consumers organize their photos (where are all the photos of Grandma?), or can be used by governments for surveillance (matching a security footage frame to a database of driver’s license photos). The classification of web users for targeted advertising also fits into this group.

Recommendation. A third type selects information to present to someone, based on their own search or purchase history, or searches performed by someone else with a similar profile to them. These systems are called recommender systems. They’re behind the ordering of your feed in social media websites, Amazon product recommendations, or movie suggestions on Netflix.

Transcription/ Translation. The fourth type is the automatic translation of information from one format to another: automatic transcription (sometimes called “automatic speech recognition” or “speech to text”), finding words and characters in images (like automatically reading license plates), machine translation of one language to another, or something like image style transfer (taking a selfie and making it look like an anime character).

and Image Generation. Then finally there’s a type that’s been very much in everyone’s mind recently: so-called generative AI or, more aptly, synthetic media machines. These are systems like ChatGPT, Gemini, or DALL-E that allow users to generate images or plausible-sounding text based on textual prompts. A “prompt”, in generative AI terminology, is the words used to describe the desired output.

That all changed in the 2010s, when one particular approach to pattern matching at scale—called “deep learning”—became practical for the first time. This wasn’t because of any magic or quantum leap in technology, but for the most part followed from innovation predicated on the falling costs of microchips and the abundance of digitized data on the web, easily accessible through a small set of platforms that centralized data sharing (Flickr, Tumblr, Google, and the like).

Hype is the aggrandizement of some person, artifact, technology, or technique that you, the consumer, absolutely need to buy or invest in as early as possible, lest you miss out on entertainment or pleasure, monetary reward, return on investment, or market share.

Like other kinds of hype, AI hype plays on FOMO (the fear of missing out): it is the repeated message that a set of technologies—currently, a set of statistical methods developed within computer science and engineering—will change the world and you, the consumer or corporate manager, absolutely must use it, lest you be left in the dust.

AI hype in particular plays a cultural function as well. It connects a commercial goal with a popular fantasy of sentient machines.

just because the hype is ungrounded in the real world doesn’t mean the hype itself doesn’t impact the world, culturally, economically, and environmentally.

the forerunners of this new field were concerned with translating dynamics of power and control into machine-readable formulations.

The murky, unethical funding networks—through unfettered weapons manufacturing then, and with the addition of ballooning speculative venture capital investments now—around AI continue to this day. So does the drawing of false equivalences between the human brain and the calculating capabilities of machines. Claiming such false equivalences inspires awe, which, it turns out, can be used to reel in boatloads of money from investors whipped into a FOMO frenzy.

By the second quarter of 2024, venture capital was dedicating $ 27.1 billion, or nearly half of their quarterly investments, to AI and machine learning companies. 33

sensible use cases are swamped by promises of machines that can effectively do magic, leading users to rely on them for information, decision-making, or cost savings—often to their detriment or to the detriment of others.

the general public has much more experience with computers than they did in the 1970s, we are also up against text extruding machines that are not only far more versatile than ELIZA, but backed by companies and investors with a deep financial interest in people perceiving their technology as a pervasive and all-powerful foregone conclusion.

a class of algorithms called “neural nets”. Neural nets are composites of mathematical functions called “perceptrons” that each take in multiple inputs and then run a calculation to determine what value to output, based on those inputs. The perceptrons are connected in a network, such that the output of each can serve as the input to many others and each of those connections is associated with a “weight”, which can be interpreted as the strength of the influence of one on the next perceptron in the network. “Neural network” is an impressive-sounding but very misleading term: they are named as such because perceptrons were very loosely inspired by a 1940s understanding of how neurons work9 in the human brain.

neural nets are a kind of supervised machine learning algorithm: in order to set the weights in the network, the training setup requires large amounts of data with the correct answer given.

the way they allow for words to be represented not by the strings of characters used to write them, but in terms of what other words they co-occur with. These so-called “embeddings” mean that similar words (like cat, dog, rabbit, hamster, and other words for pets, or run, skip, sashay, and other words for movement) are given similar representations, despite being spelled distinctly.

A key early neural language model was the BERT system, developed at Google for English in 2018. The initial BERT model13 had 340 million parameters (weights on connections between perceptrons) and was trained on 3.3 billion words of text. 14 These aren’t small numbers, but they have already been absolutely dwarfed. In July 2024, Meta released Llama 3.1,15 which has 405 billion parameters and was trained on over 15 trillion words of text.

It is well established that there’s no such thing as a corpus of texts free from bias, nor is it possible to fully prevent biased or hateful output. But that doesn’t mean it can’t be made less bad. One technique for doing this (and the one that OpenAI applied in developing ChatGPT) is called “reinforcement learning from human feedback” (RLHF), where people are employed (usually precariously and for low pay) to rate the output of the system. These ratings are fed back in, “reinforcing” better outputs and down-rating worse ones, effectively adding a layer of polish to the magic trick. Now the system’s task isn’t just to pick a word that’s likely to come next, but one that is likely to come next and receive a high rating from a human rater.

we use the words and syntactic structures we perceive as a very rich clue18 to figuring out what the person who uttered them might have been trying to get us to understand.

Simply modeling the distribution of words in text provides no access to meaning, nothing from which to deduce communicative intent. Language models thus represent nothing more than extensive information about what sets of words are similar and what words are likely to appear in what contexts. While this isn’t meaning or understanding, it is enough to produce plausible synthetic text, on just about any topic imaginable, which turns out to be quite dangerous: we encounter text that looks just like something a person might have said and reflexively interpret it, through our usual process of imagining a mind behind the text. But there is no mind there, and we need to be conscientious to let go of that imaginary mind we have constructed.

language corpora are forced through complicated machinery to produce a product that looks like communicative language, but without any intent or thinking mind behind it.

How our mind processes language can be contrasted with how we process the outputs of text-to-image models like Midjourney and DALL-E. These synthetic image machines share a lot of properties with the synthetic text machines: their function is predicated on massive data theft and profligate energy use, it’s easy to be impressed by them, and they are being used to threaten people’s livelihoods. But no one is suggesting that they are sentient—we can interpret their output (images) without imagining a mind selecting symbols in an attempt to communicate. 22

the phrase stochastic parrots, which Emily and coauthors used to make vivid how language models only manipulate the form of language, with neither understanding nor communicative intent.

AI hype reduces the human condition to one of computability, quantification, and rationality.

the project of identifying general intelligence is inherently racist and ableist to its core, making the project of chasing artificial general intelligence foolhardy at best, and deceptive and dangerous at worst.

General intelligence is not something that can be measured, but the force of such a promise has been used to justify racial, gender, and class inequality for more than a century.

There’s real money invested in this work, much of it coming from venture capitalists. A lot of this might just be venture capitalists (VCs) following fashion, but there are also a number of AGI true believers in this mix, and some of them have money to burn. These ideological billionaires—among them Elon Musk and Marc Andreessen—are helping to set the agenda of creating AGI and financially backing, if not outright proselytizing, a modern-day eugenics. This is built on the combination of conservative politics, an obsession with pro-birth policies, and a right-wing attack on multiculturalism and diversity, all hidden behind a façade of technological progress.

Silicon Valley has deep and enduring connections to eugenicist thought. Stanford University’s first president, David Jordan Starr, was a strident eugenicist who recruited eugenicist thinkers to major professorships within the university, among them Lewis Terman. 35 Terman, the codifier of the Stanford-Binet IQ test, adapted Binet’s original cognitive test36 in order to promote racial hierarchies that put white people at the top. Today, the kingmakers in the Valley have sometimes exuberantly, sometimes quietly, endorsed and financially supported eugenicist thinkers and alt-right politicians. These are the same few who have the power to make or break newcomers in the crowded artificial intelligence market.

for corporations and venture capitalists, the appeal of AI is not that it is sentient or technologically revolutionary, but that it promises to make the jobs of huge swaths of labor redundant and unnecessary. Corporate executives in nearly every industry and mega margin-maximizing consultancies2 like McKinsey, BlackRock, and Deloitte want to “increase productivity” with AI, which is consultant-speak for replacing labor with technology.

Automation has always been part of a larger strategy of shifting costs onto workers and accruing wealth for those in control of the machines.

Luddites were instead against technologies of control and coercion, and concerned about the loss of jobs, health, and community. 8 Even though what the Luddites fought against would easily be recognized as “automation”, the word wasn’t invented until the late 1940s. We owe the word “automation” in its modern sense to Delmar S. Harder, a vice president at Ford, who reportedly coined it in 1948.9

Modern logistics and information economies are built on automation, surveillance, and the reduction of people into mere objects, the grease on the gears. AI is part of a longer tradition within global industry of finding ways to replace labor, and/ or enforce grueling schedules and working conditions in the name of productivity.

Google’s novel search algorithm seemed to outpace similar services offered by AltaVista and other early search engines. But now Google Search itself structures the web, and not in a way that benefits the broader public: 22 Google is first and foremost in the business of selling ads, 23 not providing helpful access to information. Search engine optimization (SEO) consultants can extort high fees with promises to get their clients’ sites to the top of Google’s results by selecting keywords and optimizing web pages, which leads Google Search to prioritize some pages over others for generating ad revenue.

even as so-called AI is being used to displace workers, especially relatively well-paid workers in wealthy countries, it turns out that tools that we are told are fully automated are not automated at all. Instead, they are powered by a great deal of labor, which is hidden behind computerized interfaces and kept out of sight of users to maintain the illusion of automation.

Given that media synthesis machines recombine internet content into plausible-sounding text and legible images, companies require a screening process to prevent their users from seeing the worst of what the web has to offer.

This industry has been called by many names: “crowdwork”, “data labor”, or “ghost work” (as the labor often goes unattended and unseen by consumers in the West).

Amazon’s Mechanical Turk, a system for the buying and selling of labor for performing small sets of online tasks.

Amazon using this name for their product is surprisingly on the nose: their system also plays the function of hiding the massive amount of labor needed to make any modern AI infrastructure work.

ImageNet’s pattern of exploiting low-paid workers around the world has become the industry norm in artificial intelligence (in addition to indiscriminate scraping of images and text from the web,

When executives are threatening to replace your job with AI tools, they are implicitly threatening to replace you with stolen data and the labor of overworked, traumatized workers making a tiny fraction of your salary.

Another role arising due to the generative AI rush is the “red-teamer”. Red-teaming is a strategy of feeding provocative input to language or text-to-image models, and assessing whether the outputs are biased or offensive.

In the tradition of the original Luddites, writers, actors, hotline workers, visual artists, and crowdworkers alike show us that automation is not a suitable replacement for their labor. We don’t have to accept a reorganization of the workplace that puts automation at the center, with devalued human workers propping it up.

venture capitalists and others seeking to cash in will run the AI con to disconnect the rest of us from social services, promoting a drive for scale that renders humane and connected services impossible.

Unfortunately, that chatbot isn’t able to reliably retrieve and convey accurate information; like all LLM-based chatbots, it was designed to make shit up.

a rat’s nest of confused science feeding into public relations campaigns and hype-tastic misapplication of poorly matched technologies. In other words, just exactly the kind of ouroboros of AI hype that is our central focus.

With the excitement about large-scale image and language models in the mid to late 2010s, however, evaluation in research started to go right off the rails, when researchers started creating benchmarks which they claimed tested for things like general-purpose natural language understanding. Once the benchmark is published, it turns into a contest for developers to compete in and brag about their scores on a public “leaderboard”, which further shifts incentives away from testing models for realistic situations and towards achieving a high score on a fixed evaluation task. 50

corporations, startups, and industry-funded labs looking to claim that they have developed an “AI” for medicine or legal services, turn to standardized professional licensure exams, such as medical licensing exams and state bar exams. They do this instead of getting down to specifics about how the system is meant to be used and evaluating it in that context. This would be laughable if it weren’t so alarming: What would society need with a system that takes standardized tests? These evaluations tell us little about how such systems would perform on a particular legal or medically oriented task. All they’re really good for is hype-filled headlines like “ChatGPT Passes Bar Exam”, 54 reinforcing the misconception that reciting the correct forms is all that is needed for practicing law, medicine, therapy, and the like.

The use of standardized or professional tests or other artificial tasks in the evaluation of AI systems is a giant red flag. It typically signifies a cartoon understanding of the work that AI boosters claim their system can do; a disregard for the creativity, person-to-person connection, and care involved in the jobs they claim to replace; and a callous willingness to fob off anyone who might be dependent on the social safety net onto automated facsimiles of the services that society owes them.

if you have the misfortune of having had your health care provider bought up by Amazon, your access to that provider may now by denied by a stochastic parrot.

Everything about Hippocratic AI is appalling. Shah seems to have missed that empathy and personal interest both require subjective experience and human connection. Keeping a transcript of every (conveniently already digitized) interaction isn’t remembering conversations, but it surely is building up a trove of data for future monetization.

these tools can’t actually have empathy. They can repeat a set of words that, together, can be interpreted as empathy. But they cannot feel feelings nor recognize them in others. They cannot relate to us about the human condition,

despite what Google or digital health executives say, the push for AI in health care won’t broaden access. What it will do is worsen the working conditions of nurses and other health care providers, while widening the gulf between those who can get quality health care (which will remain provided by humans) and the rest of us (who will be left with cheap electronic knockoffs).

large language models can extrude text on any subject. Input a set of symptoms, and what comes out looks like a diagnosis. Input a legal query and what comes out looks like a contract or legal brief. Input a school subject and request for a lesson plan on literally anything and what comes out will look like a set of facts that you can teach students and exercises to have them do.

when someone suggests a robo-doctor, robo-therapist, or robo-teacher, we should ask: Why isn’t there enough money for public clinics, mental health counseling, and schools? Text synthesis machines can’t fill holes in the social fabric. We need people, political will, and resources.

Automating what is rote or describable via algorithm is just engineering, but forging something capable of creativity demonstrates a quantum leap into a technology that can approximate activity that had heretofore been the sole domain of human beings.

our current moment of AI hype features overblown claims of mathy maths that are capable of producing art, science, and journalism, three fields of endeavor that are creative at their core, but are often experienced via textual, visual, or other artifacts.

Like all generative models that use machine learning, a model can only generate media that is a weakly remixed version of what is in its training data. This means that the training data strongly determines what ends up in the model.

search engine optimization (also known as SEO). That is, it’s become a strategy to get users to click on particular pages that appear in search results (which means, mostly on Google), either for ad revenue or e-commerce.

The major functions of art include sharing experiences and providing insight into the human condition—not to mention the joy and fulfillment of artistic expression. As philosopher and technology scholar Johnathan Flowers has said20, the purpose of art is to signal a particular kind of intention and to convey a particular type of experience, and this is precisely what AI art lacks. Art exists as a means to convey something about the human condition. A diversity of art forms exist because we, as humans, are diverse. An expression of human experience can be simple or complex but need not involve a high level of technical acumen. By this measure, we find the claims that crafting prompts is akin to “democratizing” art and producing the same joy to be unconvincing.

many defenders of AI art have argued that when humans make art we are also always “just” remixing ideas from other artworks, such that the “borrowing” (more accurately: stealing) from artists like Ortiz is justified. But there is an enormous difference between the practice of craft and the practice of writing a successful prompt: when we reference or remix ideas from other artwork, we are drawing on both the form and meaning of the art. We pull in the form because it was meaningful to us and we want to invoke that meaning in our creation.

Like academic scholarship, which we discuss below, artistic practice is a social activity, one that is ostensibly performed with a respect towards others’ prior work, since they are peer creators. Even in work that is meant as a critique, creators know who the target is, or center on one or two paradigmatic examples of that artistic style. In an academic context, we would call this citational practice. Citational practice22 is an acknowledgment of what came before and that you were not the first person to develop an idea. Citation also operates as a currency in status-based fields like art and academia. When people using synthetic media machines generate books, images, or other media, there is no citational practice or acknowledgment of the social production of the work. It’s just a cheap rip-off.

Much of the conversation24 around copyright—at least in the U.S.—hinges on whether the use of copyrighted works can be used in the creation of derivative works, what is known as the “fair use exception” to copyright law. Existing case law has established a four-factor test of “fair use”: whether the work is sufficiently “transformative,” the nature of the work (e.g., was it published or unpublished?), the amount and substantiality of the portion taken, and the effects of the resultant work on the potential market. Derivative works must meet all four factors to pass this test in order not to be in violation of copyright.

Their argument goes something like this: model training involves copying the original work, yes, but then it only focuses on the words (for texts) or on the pixels (for images), turns them into numbers for input into models, and then outputs something different altogether.

Venture capital firm Andreessen Horowitz warned27 that all of their investments in AI would be worth a lot less if they had to abide by copyright law: “Imposing the cost of actual or potential copyright liability on the creators of AI models will either kill or significantly hamper their development.” That is, if they actually had to pay artists, illustrators, and writers what their content is worth, rather than simply stealing that content from the web, their business model would fall apart.

with LLMs, the situation is even worse than garbage in/ garbage out—they will make papier-mâché out of their training data, mushing it up and remixing it into new forms that don’t preserve the communicative intent of original data. Papier-mâché made out of good data is still papier-mâché. 29

LLMs are simply not suited to the task of synthesizing and presenting scientific information.

Reviewers consider whether the papers are appropriately grounded in previous work, use methods that make sense for the research questions they are addressing, have collected relevant data, and include convincing argumentation that draws on the data to reach conclusions about the research questions.

AI boosters have suggested that the process of peer review could be sped up with the judicious application of LLMs. Perhaps, they say, the chatbots could write a first draft of the review or suggest possible problems with the papers being reviewed! This isn’t hypothetical: researchers at Stanford studied39 peer reviews of papers submitted to conferences about natural language processing, machine learning, and robot learning from 2020 to early 2024 and found that between 6.5 and 16.9 percent of the peer reviews written after the release of ChatGPT contained text likely to have been either simply the output of an LLM or substantially modified by one—a sharp increase compared to pre-ChatGPT.

The absurdist writer Douglas Adams caricatured this kind of wishful thinking perfectly in the late 1970s, with the characters in The Hitchhiker’s Guide to the Galaxy who developed a supercomputer to give them the ultimate answer to the ultimate question of life, the universe, and everything. That answer, they learned after generations of waiting, was 42. Of course, such an answer is useless without the corresponding question, and their supercomputer wasn’t powerful enough to determine the question. It was powerful enough, however, to design an even bigger computer (the planet Earth, as it happens) that could, given 10 million years, calculate the question. We can’t delegate science to machines, because science isn’t a collection of answers. It’s a set of processes and ways of knowing.

An understanding of science behind an imagined autonomous AI Scientist (stylized with a capital S, following Nobel Turing Challenge marketing copy) is markedly different from how science actually happens. In this view, scientific knowledge is simply made up of collections of empirical facts, which are to be found through technical processes that just need to be refined enough. It follows that if we could just get more of those facts more quickly, we’d be benefitting from more science and more technology. However, this view leaves no room for understanding science as a fundamentally human and social activity, that can only take place at a human scale, through communication among scientists, and between scientists and the broader public. As with AI “art” discussed above, AI boosters think that science is only about ideas, rather than communities of practice.

The climate crisis is fundamentally a collection of social problems, about building political will to overcome current economic incentives and about how to allocate resources to accommodate climate refugees. We can’t technology our way out of it—and neither could a hypothetical AI scientist.

the imagined tools represent the epitome of a view from nowhere, 46 or the idea that one can have objective knowledge of a set of truths, uncolored by their personal experience.

With each new university press release about AI and each new announcement of grant opportunities around the “potential” of AI for benefiting science, we see the AI fashion trend becoming ever more all-encompassing. One thing that can help resist the trend is keeping a clear view of what’s really driving AI-in-science: venture capital and Big Tech. The research labs in industry present themselves as doing fundamental research, aiming to produce knowledge that benefits humanity in general and giving back to the scientific community in particular. For example, the marketing copy for projects like Google DeepMind’s research on crystal materials48 is rife with allusions to the potential benefits for such important causes as better solar panels. But they actually aren’t as beneficial to the scientists working in the relevant fields as the advertising copy would have

when authors turn to LLMs for prose when they are in a hurry, they may not be in a position to check it for accuracy, especially if it sounds convincing and/ or uses words or turns of phrase outside their own linguistic competence. This is especially true in the literature review case: authors taking that shortcut almost certainly don’t have time to check the papers being cited (which furthermore might not exist), let alone check for what they should have cited but didn’t.

Regardless of why they are produced, synthetic or partially synthetic scientific papers damage the scholarly information ecosystem, mixing unreliable texts that no one can really vouch for in among those that, in theory, other scholars could be learning from and building on.

An example of a successful use56 comes from Nikhil Garg and colleagues, who used LLMs trained on text collections of American English from different decades, starting in the 1910s, and observed how words related to different genders, for example, have clustered with different words over time. With this method, they track changes in gender and ethnic stereotypes in English text across the twentieth century. This research succeeds because they are treating the LLMs as exactly what they are: representations of patterns of language use in their training texts. These tools are useful to have in one’s toolkit. But it’s not a scientific revolution, nor a solution to a whole scientific field. And it certainly isn’t a reason to abandon other approaches to science and jump on the AI bandwagon.

Science is squarely in the hype danger zone.

the peer review crisis also has close ties to the further casualization and privatization of the university, including the reduction of tenure-track lines and the massive increase in classes being taught by adjunct and part-time faculty.

The drive to adopt AI is, of course, part of a much longer story about the decline of quality journalism across the world, driven by the dramatic reduction of advertising revenues, the consolidation of media companies, and the loss of trust in media as an institution.

both the creation and output of these systems is damaging—to individual creators, science, and reporters, and to the larger ecosystems in which they are enmeshed. Today’s synthetic media extruding machines are all based on data theft and labor exploitation,

The use of these systems does further damage socially: displacing working artists and journalists, warping the practice of science, and polluting the information ecosystem. And their existence undermines the position and value of craft across these endeavors.

a set of end-of-times tech visions that have different monikers: “p( doom)”, “existential risk” (or “x-risk”), or, more recently, “critical harm” 6 (as proposed in California legislation by friend-of-the-tech-industry state senator Scott Wiener).

From the Doomer/ Booster point of view, then, it is of critical importance to work out how to make sure that the supposed coming AI overlords are “aligned” with humanity’s goals.

a peculiar idea called “alignment”. As the Doomer notion of risk is rooted in fantasies about sentient AIs going rogue, rather than anything people might do with automation, a primary focus of AI safety research is based on the idea that the solution is to design AI systems where are “aligned” with “human values”. A popular formulation, from a book authored by commentator Brian Christian, has named this the “alignment problem” 8: “how to ensure that these models capture our norms and values, understand what we mean or intend, and, above all, do what we want.” The alignment problem is considered the crown jewel in a panoply of problems for those involved in AI safety research. Those who engage in this work portray it as virtuous, in contrast to seeking AI development solely for profit’s sake. Alignment is supposedly about having some faith in humanity and directing our energies such that AI will be beneficial for everybody.

When AI Doomers warn against existential risk, what they really mean is “existential risk for well-off, white, Western, and able-bodied people who are insulated from becoming climate refugees.”

old wine in a new wineskin: a true belief that rampant capitalism is the solution to society’s ills, a new picture frame around the California Ideology’s28 social liberalism with unfettered markets.

TESCREAL stands for Transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism, and Longtermism. While all of these have their own tenets, they have overlapping sets of adherents and can be summed up in a few words. Those in the “TESC” part of the acronym maintain that humans are, at some point, going to merge with machines and then fly off Earth to colonize space. The “REAL” part of the acronym concerns an ultra-utilitarian ethic that holds that we, as humans, need to optimize our behavior to do the most good with the resources available to us.

the faulty logic of many strains of development economics33 to fund technical interventions that, at best, save very few lives, and at worst reorient local incentive structures towards environmental ruin and corruption.

The technologies in question are not the equivalent of nuclear weapons, nor even of nuclear reactors. They are more like a panopticon40—philosopher Jeremy Bentham’s invention that allows a single prison warden to keep track of hundreds of prisoners at once. Or like the surveillance dragnets that track marginalized groups in the West and people on the move escaping dire conditions in their countries of origin. Or perhaps they are a toxic waste, salting the earth of a Superfund site. They are also a scabbing worker, crossing a picket line at the behest of an employer who wants to signal to the picketers that they are disposable. The totality of systems sold as AI are these things, rolled into one. The danger is not from some hypothetical extinction-level event. The danger emerges from rampant financial speculation, the degradation of informational trust and environments, the normalization of data theft and exploitation, and the data harmonization systems that punish the people who have the least power in our society by tracking them through pervasive policing systems.

With no clear definition of what they’re trying to measure, much less a solid foundation for claiming that they are actually measuring it, the Doomers and the Boosters are left to fall back on obfuscation and appeals to awe.

The actual power-hungry computers are conveniently hidden in the fluffy, harmless-sounding metaphor of “cloud computing”, which information science researchers Alan Borning, Batya Friedman, and Nick Logler53 point out is at direct odds with the actual materiality of these systems. When algorithm developers train a large statistical model, or when users submit a prompt to ChatGPT, it’s easy to imagine the processing happening in some abstract, virtual space. When the work is done on remote servers, we don’t even have to hear the fans keeping them cool! But in fact, everything about cloud computing is environmentally intensive: 54 the mining of metals and minerals required as raw materials, the use of large amounts of PFAS (“ forever chemicals”) in the production of microchips, the energy required to both make hardware in chip fabrication plants and run the systems, the water used to keep data centers cool, and the e-waste produced as each generation of machines is retired in favor of the next.

The “AI Overviews” feature that Google added to search results in 2024 likely consumes 30 times more energy per query than just returning links66. This feature was enabled by default, without users even having to invoke it.

While we don’t know how much of that data center usage can be attributed to AI systems (as opposed to other things like video-streaming services), it’s telling that68 energy demands for generative AI systems globally are expected to increase dramatically, rising tenfold between 2023 and 2026, and that Google, Microsoft, and OpenAI are putting billions of dollars into new data centers.

The more we can pierce the cultural bubble that Sam Altman and his kind live in, the better we can upset the idea that the encroaching of these systems into every area of life is inevitable. Synthetic media is cheap and tacky. Let them know.

text synthesis machines are a terrible match for this use case, on two levels. As we’ve seen, they’re inherently unreliable, being designed to make shit up. But it’s the second level that we want to address here: friction in information access is actually not only beneficial, but critically important.

The friction that chatbot-based information access systems are trying to reduce or remove includes the work of deciding which of the links returned by a search engine has the information sought and where in the website the link points to can that information be found. Doing that work is important to building our understanding of the information landscape. Scanning a set of links gives us information about what information sources are available, and, if there are some we are expecting but don’t see, we have the opportunity to refine our search.

Finding contradictory answers on different pages—and, crucially, knowing the source of each—allows us to learn what kinds of knowledge are contested, who is doing the contesting, and how each of those sources fits into our own positions.

If we don’t resist this onslaught of synthetic text, the consequences are going to be bad, not just for our own information literacy, but also for the information ecosystem, and doubly so. There are the first-order effects of synthetic media spilling into the information ecosystem, and second-order effects of lower levels of information literacy impeding our collective ability to tend to and maintain that ecosystem.

already have a good deal of regulatory tools to deal with the oversized claims, monopolistic practices, and other corporate malfeasance of AI boosters. In the U.S., regulators who handle consumer complaints, communications infrastructure, civil rights, and labor issues have kept a keen eye out for hyped-up snake oil that companies are slinging. In April 2023, the Federal Trade Commission and several other federal agencies22 announced that there is no AI loophole—that is, companies producing generative AI must abide by current consumer and nondiscrimination rules on the books. These agencies added that their role is to regulate the actions of businesses, regardless of whether those businesses use automation.

“Your therapy bots aren’t licensed psychologists, your AI girlfriends are neither girls nor friends, your griefbots have no soul, and your AI copilots are not gods.”

we’ve been subjected to accelerating usage of AI as a pretext to surveil, arrest, and deport people; accelerating environmental impact of data centers to run the AI systems; and hundreds of car crashes, including at least seventeen fatal ones, as innocent bystanders are subjected to informal beta tests of Tesla’s misleadingly advertised “Full Self-Driving” technology.